A note on new trends in data-aware scheduling and resource provisioning in modern HPC systems

The Big Data era [1,2] poses new challenges as well as significant opportunities for High-Performance Computing (HPC) systems such as how to efficiently turn massively large data into valuable information and meaningful knowledge? It is clear that computationally optimized new data-driven HPC techniques are required for processing Big Data in rapidly-increasing number of applications, such as Life Sciences, Particle Physics and Socioeconomical systems. The realm of HPC systems lies in sharing of the ‘‘multicore’’ hardware resources among the software applications. Key characteristics of HPC systems include high processor density, high speed Input/Output (IO), and high-density cooling techniques. In the Pre Grid computing era (before 2000), the HPC was always exclusively referred to as ‘‘supercomputing’’. In grid-based HPC era, the Globus project in conjunction with cluster job scheduling systems such as Portable Batch System (PBS), and Platform LSF (recently acquired by IBM) has dominated the middleware research efforts and powered numerous computational grids across the globe. PBS and Platform LSF implemented scheduling techniques and cluster resource provisioning mechanism for allocation of available compute resources to HPC applications. However, with the emergence of modern HPC systems (e.g., Amazon EC2 Cluster CPU Instances, Univa Grid Engine, IBM HPC Cloud, etc.) powered by cloud computing and virtualisation technologies, the job scheduling techniques implemented by traditional HPC schedulers (e.g. Platform LSF, PBS) are facing serious limitations. The main reason for this state of affairs is that modern HPC systems cannot support on-demand scalability, strong performance guarantees, and improved fault-tolerance, which the traditional HPC scheduling techniques are unable to cater for or take advantage of. In reality, the traditional HPC schedulers were not designed for the cloud computing and virtualization era. Hence, this special issue solicits papers related to topics including techniques for optimizing the performance of traditional HPC applications on new ‘‘multi-core’’ cloud systems, novel extensions to traditional HPC schedulers for Big Data application scheduling, dynamic resource provisioning for HPC applications on the cloud, techniques for optimizing HPC application specific performance and energy constraints, workflow scheduling techniques, and so on. The Call for this Special Issue received a number of submissions. After a two-phase review process we accepted eight papers of very high quality. This includes one research survey paper that gives an interesting overview of the current big data problems and solutions. The other papers are related to scheduling strategies on Cloud-related platforms with individual scheduling goals.